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Abstract 

This study examined the effectiveness of three types of think aloud methods for understanding 
elementary teachers’ cognitive processes as they used a professional development website. A 
methodology combining a retrospective think aloud procedure with screen capture technology 
(referred to as the virtual revisit) was compared with concurrent and retrospective think aloud 
procedures. Elementary teachers from a large metropolitan area were assigned to one of the 
three think aloud conditions (N = 45). Participants in the concurrent condition verbalized their 
thoughts while simultaneously navigating a professional development website for 20 minutes. 
Participants in the retrospective condition verbalized their thoughts following their 20-minute 
website navigation without any aids. Finally, participants in the virtual revisit condition 
verbalized their thoughts while viewing a screen recording of their website navigation. Think 
aloud protocols were analyzed to detennine the frequency of cognitive processes verbalized by 
participants in each condition. The findings of this study indicated significant differences in the 
types of verbalizations produced by participants across the three think aloud conditions. In 
addition, findings reveal benefits and limitations of employing each type of think aloud method 
in the context of a professional development website. 
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Introduction 


Elementary teachers are a necessary foundation for building successful programs in the 
classroom (Gambrell & Anders Mazzoni, 1999; Pressley, Mohan, Raphael, & Fingeret, 2007). 
Successful programs begin with a repertoire of pedagogical knowledge, pedagogical content 
knowledge, and research-based instructional practices. This repertoire of infonnation can be 
delivered to practicing elementary teachers through various professional development 
opportunities (Cervetti, Kulikowich, Drummond, & Billman, 2012; Desimone, 2009; Kao, Wu, 
& Tsai, 2011). 

One facet of teacher professional development is online learning, which occurs when 
professional knowledge is constructed from multiple modes of digital infonnation—photographs, 
videos, and interactive tools, to name a few (Mayer, 2002). Online learning is a favored 
approach to professional development because it creates accessible opportunities; online learning 
takes place within platforms that deliver infonnation in a means that removes time, place, and 
situational baniers (Kanuka & Nocente, 2003). Online learning opportunities have also been 
shown to have positive effects on and even change teachers’ pedagogical and content knowledge, 
classroom practice, and student outcomes (Weschke & Barclay, 2011). As elementary teachers 
increasingly turn to the Internet for their professional learning (Charalambousa & Ioannou, 2011; 
Kao et ah, 2011), it is essential to examine how they use and learn from online resources and 
professional development websites. 

Most studies that have examined online teacher learning have gathered data through 
surveys, questionnaires, and interviews (Duncan-Howell, 2010; Hur & Brush, 2009; Kao et ah, 
2011). These methods offer information about teachers’ attitudes towards online professional 
learning; however, data generated from these methods is limited to participants’ recollection of 
past events. A method that tracks teachers’ cognitive processes as they make online choices is 
necessary to provide further insight into how teachers use and leam from online environments. 
The think aloud methodology is an approach that can track teachers’ cognitive processes during 
decision-making activities. While think aloud research is extensive, studies that compare the 
effectiveness of different think aloud methodologies for understanding teachers’ cognitive 
processes as they navigate online resources are limited (Kuusela & Paul, 2000; van Gog, Paas, 
van Marrienboer, & Witte, 2005). The purpose of this comparative study was to examine the 
effectiveness of three types of think aloud methods for understanding elementary teachers’ 
cognitive processes as they used a professional development website. A methodology 
combining a retrospective think aloud procedure with screen capture technology (referred to as 
the virtual revisit) was compared with concurrent and retrospective think aloud procedures. 

A detailed discussion of the think aloud methodology sets the foundation for this paper. 
The current study’s methods are then presented, followed by the results and a discussion of the 
significant findings. The study’s limitations and educational implications conclude this paper. 

Literature Review 

Thinking aloud has historical roots in introspection analysis, a fonn of data collection 
aimed at investigating psychological claims and theories of mind during the eighteenth century 
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(Boren & Ramey, 2000; Ericsson, 2002). The cognitive revolution of the 50s and 60s produced 
alternative types of verbal reports of thinking to gather information about cognitive structures 
and processes (Ericsson, 2003). Today, the think aloud method most widely employed is based 
on the techniques of protocol analysis by Ericsson and Simon (1984, 1993). As described by 
Ericsson and Simon (1984), thinking aloud captures cognitive processes in real time and verbal 
reports “provide the most infonnative data available on thinking during cognitive tasks” 
(Ericsson, 2003). Cognitive processes underlying decisions and behaviors are usually “hidden 
from direct observation” (Gaissmaier, Fife, & Rieskany, 2010, p. 141). However, the think 
aloud method makes monitoring cognitive processes possible—the think aloud generates direct 
data about the ongoing cognitive processes that occur during task perfonnance (Jaspers, Steen, 
van den Bos, & Geenen, 2004). 

Ericsson and Simon (1984) describe three levels of verbalizations that can occur during 
the think aloud method. The first two levels require infonnation processing in the participant’s 
short tenn memory and the third level requires additional cognitive resources and retrieval of 
information from long term memory (Olmsted-Hawala, Murphy, & Hawala, 2010). While 
Ericsson and Simon (1984) state that Level 3 verbalizations or higher cognitive processes are 
less reliable because they involve access to long-tenn memory, usability researchers suggest that 
this type of data provides useful infonnation about online learning, website user goals, and 
online behaviors (Boren & Ramey, 2000; Guan, Lee, Cuddihy, & Ramey, 2006; Olmsted- 
Hawala et ah, 2010). 

Usability researchers most often employ the concurrent and retrospective think aloud 
methods to gain insight into web seeking behaviors and to evaluate a website’s content and ease 
of use (Aranyi, Schaik, & Barker, 2012; Barzilai & Zohar, 2012; Branch, 2006; Kuusela & Paul, 
2000). During the concurrent procedure participants verbalize their thoughts aloud while they 
simultaneously complete a task. Verbal reports that result from the concurrent procedure 
generate data about the website user’s navigational experience. For instance, Aranyi and his 
colleagues (2012) conducted an exploratory study of interaction experience with a news website. 
The concurrent think aloud yielded five categories of experience based on the participants’ 
evaluative statements: impression, content, layout, infonnation, architecture, and diversion 
(Aranyi et ah, 2012). Similarly, Barzilai and Zohar (2012) utilized the concurrent procedure to 
examine epistemic thinking in action. Data was collected to shed light on the relationship 
between sixth grade students’ knowledge construction and their online practices. Analysis 
revealed a positive relation between students’ online strategies and their epistemic cognition 
(Barzilai & Zohar, 2012). Damico and Balidon (2007) also employed the concurrent procedure 
to examine how elementary students engage with an educational website. Findings from their 
study highlight how elementary students evaluated claims and evidence of online educational 
resources (Damico & Balidon, 2007). 

The retrospective procedure is also referred to as post-task testing, retrospective report, 
and think after. Retrospective think alouds alone are used less often in the fields of online 
learning and website usability since they require participants to think aloud after a task has been 
completed. An international survey found that just 5% of think aloud studies (not limited to 
website usability) employed the retrospective technique, whereas 89% used the concurrent think 
aloud, and 6% used an alternative think aloud (McDonald, Edwards, & Zhao, 2012). 
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One of the main reasons why retrospective think alouds are used less often is due to the 
fact that the procedure relies on the ability to recall decisions after a task has been completed. 
As participants recall their decisions, infonnation may be incomplete and include errors, 
omissions, and substitutions (Branch, 2006). For instance, a comparative study that examined 
retrospective and concurrent verbal protocol analysis in the context of a decision-making task 
found retrospective reports more prone to errors of omission whereas concurrent reports 
contained more relevant infonnation about the decision making process (Kuusela & Paul, 2000). 

While retrospective procedures are limited by the fact that they may be incomplete and 
include enors, omissions, and substitutions (Branch, 2006), they have the advantage of freeing 
cognitive resources by thinking aloud after the task has been completed—retrospective think 
alouds do not interfere with task perfonnance (McDonald et ah, 2012). Concurrent think alouds, 
on the other hand, can interfere with task performance since participants verbalize their thoughts 
while they simultaneously complete a given task—participants engage in two different processes 
at the same time. When two processes occur simultaneously there is an increase in cognitive 
load—“the level of mental energy required to process a given amount of infonnation” (Ping Lim, 
2004, p. 17). As a result of a higher cognitive load, task completion may be compromised during 
the concurrent procedure and resulting think aloud reports are often procedural in nature 
(McDonald et ah, 2012; van Gog, Kester, Nievelstein, Giesbers, & Paas, 2009). Findings of 
such studies suggest that alternative think aloud methods warrant attention. 

An alternative to the concunent and retrospective procedures is the virtual revisit think 
aloud method. The virtual revisit is a variation of the retrospective think aloud method and 
allows participants to review and comment on a visual recording of how they interacted with a 
particular website. The goal of the virtual revisit is to aid recall of original events and thought 
processes by using a screen-capture recording of participants’ navigational experiences. Similar 
to cued retrospective reporting where participants are given instructions to report retrospectively 
on the basis of a record of observations (van Gog, Paas, van Marrienboer, & Witte, 2005), the 
virtual revisit think aloud combines a retrospective think aloud with screen capture technology to 
aid recall of original events and thought processes. 

Despite the limitations of the concurrent procedure, it has been widely used in usability 
research, mostly as a means to evaluate a given website—participants verbalize their thoughts 
about the ease of use and accessibility of information. While evaluative data contributes to the 
refinement of professional development websites, user experience is a complex process and 
usability research should go beyond evaluating websites to include a range of cognitive 
processes and learning strategies (Dillon, 2001); the virtual revisit think aloud has the potential 
to avoid the limitations of the concurrent and retrospective procedures. In addition, few studies 
have been undertaken to compare the relative utility of different think aloud procedures (Kuusela 
& Paul, 2000; van Gog, Paas, van Marrienboer, & Witte, 2005). The current study addresses 
these gaps in the literature by examining the utility of three think aloud methods during online 
professional learning. 

The following research questions guided the study: 

1. To what extent do participants’ verbalizations differ across the three think aloud methods? 
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2. What are the benefits and limitations of employing each type of think aloud in the context of 
online learning? 


Methods 

Study Context 

This research was undertaken within the context of the development and refinement of a 
professional development website. The website is a multimedia evidence-infonned literacy 
professional development website that provides free professional learning resources for 
elementary teachers and educators. The website is highly complex and interactive, and includes 
virtual tours of exemplary classrooms (PreK-6), video clips of expert teachers explaining and 
demonstrating effective educational practices, detailed lesson plans, photographs of teaching 
materials, exemplars of student work, and links to related research articles. 

Participants 

Forty-five practicing elementary teachers from a large metropolitan area participated in 
this research over an eight-month period. All participants completed informed consent forms 
and volunteered to participate in this study. 

Data Sources 

Demographic questionnaire. A demographic questionnaire was administered to 
participants to obtain data on a range of relevant factors based on the literature related to teacher 
development and online learning (e.g., age, gender, years of teaching experience, education, 
extent of involvement with various web-based technologies). Participants were asked to 
complete the questionnaire online prior to a one-on-one meeting. The questionnaire was 
administered through an online survey program. After participants completed the questionnaire, 
they were placed in one of three think aloud conditions (concurrent, retrospective, and virtual 
revisit). The conditions are described below. Stratified random assignment was employed to 
ensure that certain demographic features were represented within each group as equally as 
possible. Stratification variables were selected based on the literature on web navigation and 
teacher professional development. Research has found that gender (Page, Robson, & Uncles, 
2012; J. Pearson, A. Pearson, & Green, 2007), age (Laberge & Scialfa, 2005), subject matter 
knowledge and experience (Laberge & Scialfa, 2005), and computer self-efficacy (Page et ah, 
2012) influence the perception and navigation of websites; therefore, comparable aspects 
including gender, age, years of teaching experience, current teaching grade, comfort with 
technology, and frequency of Internet use for professional purposes, were selected as the key 
variables used for the group assigmnent. 

Table 1 (next page) summarizes the demographic characteristics for the participants 
across the conditions. As can be seen from the table, participants in the three groups were quite 
similar with respect to gender, age, years of teaching, current grade, comfort with technology, 
and frequency of Internet use. In other words, the stratified random assigmnent was successful. 
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Table 1 


Demographic Characteristics 


Characteristic 


Concurrent 

Condition 

( 77 = 15) 

Retrospective 

Condition 

( 77 = 15) 

Virtual 
Revisit 
Condition 
( 77 = 15) 

Total 

(N = 45) 



n (%) 

77 (%) 

77 (%) 

77 (%) 

Gender 


Male 

3 (20%) 

3 (20%) 

3 (20%) 

9 (20%) 


Female 

12 (80%) 

12 (80%) 

12 (80%) 

36 (80%) 

Age 


25-29 

3 (20%) 

3 (20%) 

3 (20%) 

9 (20%) 


30-34 

3 (20%) 

5 (33%) 

4 (27%) 

12 (27%) 


35-39 

4 (27%) 

3 (20%) 

4 (27%) 

11 (24%) 


40-44 

2 (13%) 

2(13%) 

2 (13%) 

6 (13%) 


45-49 

1 (6%) 

1 (6%) 

1 (6%) 

3 (7%) 


50-54 

2 (13%) 

0 

0 

2 (4%) 


55+ 

0 

1 (6%) 

1 (6%) 

2 (4%) 

Years 

Teaching 


2-4 years 

3 (20%) 

3 (20%) 

4 (27%) 

10 (22%) 


5-9 years 

6 (40%) 

6 (40%) 

5 (33%) 

17 (38%) 


10-14 years 

3 (20%) 

5 (33%) 

4 (27%) 

12 (27%) 


15-19 years 

1 (6%) 

0 

1 (6%) 

2 (4%) 


20-24 years 

1 (6%) 

0 

1 (6%) 

2 (4%) 


25+ years 

1 (6%) 

1 (6%) 

0 

2 (4%) 

Current 

Grade 


Kindergarten 

1 (6%) 

1 (6%) 

0 

2 (4%) 


Primary 

5 (33%) 

5 (33%) 

7 (47%) 

17 (38%) 


Junior 

7 (47%) 

6 (40%) 

6 (40%) 

19 (42%) 


Primary/Junior 

2 (13%) 

3 (20%) 

2 (13%) 

7 (16%) 

Comfort 

Using 

Internet 


Very 

15 (100%) 

13 (87%) 

13 (87%) 

41 (91%) 


Somewhat 

0 

2(13%) 

2 (13%) 

4 (9%) 


Not very 

0 

0 

0 

0 

Frequency of 

Internet Use 
for 

Professional 

Purposes 


> once/day 

5 (33%) 

3 (20%) 

6 (40%) 

14(31%) 
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Once/day 6(40%) 8(53%) 7(47%) 21(47%) 

Once/week 4(27%) 4(27%) 1(6%) 9(20%) 

Once/month 0 0 1 (6%) 1 (2%) 


Think aloud. Participants completed a think aloud during a one-on-one meeting. 

Concurrent think aloud. Participants in the concurrent condition verbalized their 
thoughts for 20 minutes while simultaneously completing a website task. 

Retrospective think aloud. Immediately following a 20-minute website task, participants 
in the retrospective condition recalled and verbalized their thought processes without any aids. 

Virtual revisit think aloud. Immediately following a 20-minute website task, participants 
in the virtual revisit condition reviewed their online choices virtually and verbalized their 
thoughts while viewing the 20-minute screen recording of their explorations. 

Screen capture technology. During participants’ navigation of the website, each visual 
step was captured with Camtasia Studio, a screen-recording computer software program 
developed by TechSmith (Uppal, 2011). 

Procedure 

One-on-One Meeting. The one-on-one meetings followed a sequence of events and 
lasted approximately 45 minutes. 

Website task and think aloud. The following website task instructions were presented 
to all participants: 

Your task is to use a professional development website as you normally would when 
seeking infonnation online for your teaching practices. 

While the website task instructions were consistent across the conditions, the think aloud 
instructions varied for each condition. Participants in the concurrent condition were given the 
think aloud instructions before they completed the website task, whereas participants in the 
retrospective and virtual revisit groups were given the think aloud instructions after they 
completed the website task. The purpose of infonning participants in the retrospective and 
virtual revisit conditions of the think aloud instructions after their navigation was to reduce 
reactivity—“influences of the verbalizations on the decision process” (Ranyard & Svenson, 
2010, p. 119)—as much as possible. The following passage outlines the think aloud instructions. 
The underlined portions state the different think aloud instructions given for each condition. 

In this study we are interested in what you think about when you explore a professional 
development website. In order to do this, I am going to ask you to think aloud 
(concurrent condition', as you explore the website; retrospective condition: about your 
exploration of the website; virtual revisit condition: while you view a recording of your 
exploration of the website). What I mean by think aloud is that I want you to tell me 
everything that you are/were thinking from the time you begin/began exploring the 
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website until the end of your exploration. I would like you to talk aloud constantly. I 
don’t want you to try to plan out what you say or try to explain to me what you are 
saying. Just act as if you are alone in the room speaking to yourself. It is most important 
that you keep talking. 

While think aloud studies most often employ a specific task, an open-ended task was 
used in this study to reflect as naturally as possible how teachers use and leam from professional 
development websites. To reduce disruption to the participants’ cognitive processes, prompts, 
redirections, and interventions were kept to a minimum during the process of verbalizing 
(Jaspers, 2009). Participants who were silent for a period of 30 seconds were only told to “keep 
talking.” This prompt was only given to one participant in the concurrent condition. 

Data Analysis 

Audio recordings were transcribed verbatim resulting in 45 think aloud transcripts. 

Word counts were calculated for each of the three conditions (concurrent, retrospective, and 
virtual revisit). As shown in Table 2 there are clear differences in the average number of words 
participants generated in the three think aloud conditions. 

Table 2 

Average Word Counts across the Three Conditions 



Concurrent 

Condition 

Retrospective 

Condition 

Virtual 

Revisit 

Condition 


M 

M 

M 


(SD) 

(SD) 

(SD) 

Word Count 

1676.13 

658.80 

2637.87 


(394.89) 

(229.00) 

(359.94) 


The coding scheme used to code all 45 transcripts was generated based on several studies 
relating to website usability (Aranyi et al., 2012; Cooke, 2010; Damico & Balidon, 2007; van 
Gog et al., 2005; Tan & Wei, 2006; Zhao & McDonald, 2010), teacher planning and decision 
making (Kansanen et al., 2000; Moos, 2014), and Ericsson and Simon’s levels of verbalizations 
and suggested statements (1984). The final coding scheme used to code all 45 transcripts 
includes 11 categories. Table 3 summarizes the coding scheme and offers a description of each 
category and an example from the think aloud transcripts. 

Table 3 

Coding Scheme-Categories Used to Code the Transcripts 

Verbalization Description Examples from the Transcripts 

Category 

Planning Referring to program planning, “.. .that is a lesson I can just take and 

reorganizing infonnation to fonn or tweak for my class right away.” 
develop new ideas; constructing and 
creating 
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Connecting 


Reasoning 


Reflecting 


Evaluating 

Website 

Content 


Evaluating 

User 

Experience 

Diversion 


Understanding 


Describing 

Procedural 

Behavior 


Describing 

Website 

Features 


Recalling infonnation; activating 
prior knowledge in relation to 
information presented on the site; 
finding a past example or recalling a 
concept 

Providing a rationale for making a 
navigational decision; explaining why 


Making meta-comments in reference 
to awareness of their own thinking 
and learning style 

Making judgments or expressing 
opinions about an aspect of the 
website or infonnation presented on 
the website 

Expressing a positive or negative 
feeling towards the usability and 
accessibility of the website and its 
features 

Verbalizing difficulties, including 
utterances where participants indicate 
uncertainty and confusion 

Identifying and making sense of new 
information and web-based tools 


Describing what they are doing or 
going to do or just did; statements 
about participants’ actions during 
their navigation 

Describing the spatial characteristics, 
website features and images; what 
participants notice 


“This reminds me of the mini lessons that 
I like to do at the beginning of the year.” 


“I was considering looking at text 
structures and I ended up choosing 
reading comprehension strategies because 
I’m trying to make that one of the main 
focuses of our reading program.” 

“I like to go over everything first and 
then go back and look at something more 
in-depth.” 

“This kind of photo tour really informs 
me in tenns of good practice for 
classroom management and good 
classroom environments.” 

“It’s nice that the link is already there for 
me, that I don’t have to type in a separate 
search button or go onto Google, I can 
just click onto the link.” 

“Hmm, assessment, how do I go, so what 
do I do? Enter? Search?” 

“How do I go back? This is.. .am I doing 
something wrong?” 

“At this point I’m just looking at the 
main page and I’m understanding how 
the website is organized.” 

“I’m just looking at the videos right 
now.” 


“It’s showing various pictures and 
monthly virtual tours.” 
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Reading 


“Motivation for literacy, oral language, 
knowledge building, concepts of print, 
writing conventions.” 


Reading words, phrases or sentences 
off the screen; reading out texts, 
headings, links and other on-screen 
text_ 


The categories listed in Table 3 are types of cognitive processes. Cognitive processes 
underlie the study of decision-making during professional learning experiences and can be 
categorized as higher order processes and lower order processes. Higher order processes, such as 
reasoning, involve access to “thematically related infonnation in long-term memory” (Horz & 
Schnotz, 2010, p. 238). Information stored in memory “is interrelated and rearranged, and 
extended to achieve a specific purpose” (Lewis & Smith, 1993, p. 136). Lower order processes, 
such as procedural knowledge, are normally executed in an automated way and are “only 
marginally influenced by intentional processes” (Horz & Schnotz, 2010, p. 238). In the current 
study, higher order cognitive processes refer to planning, connecting, reasoning, reflecting, and 
evaluating, whereas lower order cognitive processes refer to diversion, understanding, 
describing, and reading. The higher order and more complex cognitive processes “involve the 
manipulation of infonnation” (McLoughlin & Mynard, 2009, p. 148), whereas the lower order 
cognitive processes demand only “mechanical application of previously acquired infonnation” 
(Lewis & Smith, 1993, p. 133). 

The organization of the categories into higher and lower order cognitive processes is 
consistent with KrathwohTs revised taxonomy of educational objectives (2002). Krathwohl 
(2002) organizes six major categories and 19 sub-categories of the cognitive domain 
hierarchically and discusses how they differ in complexity. Krathwohl (2002) distinguishes 
between higher and lower order cognitive processes: the more complex categories (e.g., create) 
are higher on the scale, whereas the less complex categories (e.g., recalling) are lower on the 
scale. This distinction was taken into consideration during the development of the coding 
scheme for the current study. 

Prior to coding, the think aloud transcripts were first segmented or “unitized” into 
thought units—each utterance was deemed a separate segment or thought unit if it conveyed 
relevant infonnation and was preceded and followed by a pause and a change of ideas (Lincoln 
& Guba, 1985). According to Ericsson and Simon (1993) this procedure, in which protocols are 
unitized into phrases or segments provides more reliable findings. A second researcher was 
trained on dividing the think aloud transcripts into segments or thought units to establish inter¬ 
rater agreement. The second researcher was not involved in the research project and had no 
specific interest in the outcomes (van Someren, Barnard & Sandberg, 1994). This was necessary 
to provide results that were as objective as possible (van Someren et al., 1994). Training 
involved a review of unitizing, a demonstration of segmenting the transcripts into thought units, 
and a practice trial of segmenting a portion of one of the transcripts into thought units. As 
described by van Someren et al. (1994), during think aloud protocol analysis “coders need to be 
trained in the use of the coding scheme” and the context should be considered when interpreting 
individual phrases (p. 128). 

Following the training session, two researchers segmented 10% of the total transcripts 
into thought units. The percentage of agreements was calculated (agreements/agreements + 
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disagreements). The unitizing reliability check on 10% of 45 transcripts indicated high 
reliability, with an inter-rater agreement of 95.3% (Guetzkow's U = .012; Guetzkow, 1950). Due 
to the high inter-rater agreement and to the submission that unitizing involves subjective 
interpretation and contextualization (Lomard, Snyder-Duch, & Campanella Bracken, 2004), the 
remaining transcripts were unitized by the primary researcher, who had a thorough 
understanding of the research topic. 

Once all 45 transcripts were unitized, the total thought units were calculated. Similar to 
the differences in word counts, there are clear differences in the average number of thought units 
generated by participants in each think aloud condition (see Table 4). 

Table 4 


Average Number of Thought Units across the Three Conditions 



Concurrent 

Condition 

Retrospective 

Condition 

Virtual 

Revisit 

Condition 


M 

M 

M 


(SD) 

(SD) 

(SD) 

Number of Thought 

147.40 

30.73 

123.27 

Units 

(35.83) 

(13.15) 

(20.58) 


Similar to the unitizing procedure, a second coder who was not involved in the research 
project and had no specific interest in the outcomes was trained on the coding scheme to 
establish inter-rater agreement. Following the training session, 10% of the transcripts were 
coded by the primary researcher and second coder. Disagreements were discussed and resolved 
until an inter-rater agreement of 97% was reached. Cohen’s Kappa was performed to detennine 
consistency among raters and was found to be 0.98, Cl (0.978-0.996). Due to the high inter-rater 
agreement and the assertion that researchers themselves may serve as coders (Lombard, Snyder- 
Duch & Campanella Bracken, 2002), the remaining transcripts were coded by the primary 
researcher. 

Thought units were tallied to provide frequency counts for each category. These 
frequency counts were then transfonned into percentages based on the total number of thought 
units across categories. Coding resulted in a total of 4,521 thought units. The frequencies of 
thought unit were converted to percentages for each participant. The percentage of thought units 
in each condition were then analyzed quantitatively. Analyzing the percentage of thought units, 
as opposed to the frequency of thought units, produces a more accurate representation of the 
cognitive processes (Rosenzweig, Krawec, & Montague, 2011). This allows for a more accurate 
comparison of thought units across conditions. The word counts and total number of thought 
units were also analyzed quantitatively. Descriptive statistics, ANOVA, and Pearson 
correlations were perfonned on the proportion of thought units, total word counts, and total 
number of thought units across the three conditions (concurrent, retrospective, and virtual 
revisit). Table 5 displays the distribution of the frequency and percentage of thought units across 
the three conditions. 
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Table 5 


Frequencies and Percentages of Thought Units 


Category 

Concurrent 

Condition 

Retrospective 

Condition 

Virtual Revisit 
Condition 

Total 


Total 

% 

Total 

% 

Total 

% 

Total 

% 

Planning 

9 

0.41 

20 

3.96 

73 

4.17 

102 

2.26 

Connecting 

127 

6.04 

57 

13.23 

170 

9.08 

354 

7.83 

Reasoning 

47 

2.23 

47 

10.99 

359 

19.27 

453 

10.02 

Reflecting 

354 

16.71 

121 

26.86 

464 

25.08 

939 

20.77 

Evaluating Website 
Content 

229 

10.87 

66 

12.18 

285 

15.36 

580 

12.83 

Evaluating User 
Experience 

166 

7.96 

59 

12.13 

138 

7.47 

363 

8.03 

Diversion 

136 

5.97 

3 

0.79 

56 

2.99 

195 

4.31 

Understanding 

141 

5.99 

14 

2.97 

60 

3.36 

215 

4.76 

Describing 

Procedural Behavior 

265 

11.64 

57 

11.71 

173 

9.37 

495 

10.95 

Describing Website 
Features 

235 

10.15 

17 

3.49 

66 

3.56 

318 

7.03 

Reading 

502 

22.03 

0.00 

0.00 

5 

0.29 

507 

11.21 


Results 

A one-way between subjects ANOVA was performed to compare the percentage of each 
category verbalized by participants in each think aloud condition. The analysis revealed a 
significant main effect for planning, F{ 2, 42) = 7.05, p = .002, rf = .251; connecting, F( 2, 42) = 
8.35, p = .002, rf = .251; reasoning, F( 2, 42) = 22.01, p < .001, rf = .512; and reflecting, F( 2, 
42) = 5.36, p = .008, rf = .203. ANOVA results also indicated significant main effects for 
describing website features, F( 2, 42) =11.05, p < .001, rf = .345; and for reading, F( 2, 42) 
=43.81, p < .001, rf = .676. There were no significant main effects for the other five categories. 
Table 6 displays the summary statistics for the main effects and the means and standard 
deviations for each thought unit variable. The means indicate the average proportion of each 
type of thought unit across participants. 
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Table 6 


ANOVA Summary Statistics for Thought Units 



Concurrent 

Retrospective 

Virtual Revisit 





Condition 

Condition 

Condition 





M 

M 

M 

F 

P 

tf 


(SD) 

(SD) 

(SD) 



Planning 

0.407 (0.70) ab 

3.96 (3.83) a 

4.17 (3.66) b 

7.05 

.002* 

.251 

Connecting 

6.04 (3.59) a 

13.23 (6.53) a 

9.08 (3.81) 

8.35 

.001* 

.285 

Reasoning 

2.23 (2.46) a 

10.99 (7.65) b 

19.27 (9.16) ab 

22.01 

.000** 

.512 

Reflecting 

16.71 (7.70) ab 

26.86 (11.45) a 

25.08 (7.48) b 

5.36 

.008* 

.203 

Evaluating 

Website 

10.87 (4.96) 

12.18(8.71) 

15.36(5.27) 

1.87 

.166 

.082 

Content 







Evaluating 

User 

7.96 (6.98) 

12.13 (9.03) 

7.47 (3.30) 

2.09 

.136 

.091 

Experience 

Diversion 

5.97 (3.01) 

0.79 (2.08) 

2.99 (3.42) 

13.65 

.067 

.394 

Understanding 

5.99 (3.50) 

2.97 (4.46) 

3.36 (3.90) 

2.88 

.067 

.121 

Describing 

Procedural 

11.64 (6.21) 

11.71 (9.12) 

9.37(4.91) 

0.55 

.582 

.025 

Behavior 







Describing 

Website 

10.15 (4.87) ab 

3.45 (4.83) a 

3.56 (3.53) b 

11.05 

.000** 

.345 

Features 







Reading 

22.03 (12.79) ab 

0.00 a 

0.29 (0.64) b 

43.81 

.000** 

.676 


*p < .05, **p < .01 

a, b = significant post hoc comparisons 
Note: df= (2, 42) for all variables 


Post hoc comparisons using Tukey HSD revealed that participants in the retrospective 
and virtual revisit conditions verbalized a significantly greater proportion of planning thought 
units than participants in the concurrent condition (p < .05). In addition, participants in the 
retrospective condition verbalized a significantly greater proportion of connecting thought units 
than participants in the concurrent condition (p = .001), and participants in the virtual revisit 
condition verbalized a significantly greater proportion of reasoning thought units than 
participants in the other two conditions ip < .05). Furthermore, post hoc tests using Tukey HSD 
revealed that participants in both the retrospective and virtual revisit conditions verbalized a 
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significantly greater proportion of reflecting thought units than participants in the concurrent 
condition ip < .05). Finally, participants in the concurrent condition verbalized a significantly 
greater proportion of describing website features thought units and reading thought units than 
participants in the other two conditions ip < .05). 

With respect to the remaining thought units, findings indicated no significant differences 
between the three conditions for evaluating, diversion, understanding, and describing procedural 
behaviors. This suggests that regardless of the type of think aloud employed, participants will 
verbalize a relatively equal number of thoughts related to evaluating the website, to their 
confusion and understanding of the web-based tools and infonnation, and to descriptions of their 
own actions and online behaviors. 

A one-way between subjects ANOVA was also perfonned to compare the frequency of 
words and thought units verbalized in each think aloud condition. The analysis revealed a 
significant main effect for both variables across the three conditions: F{2, 42) = 130.42, p < .001, 
q 2 = .861 for word count; and F( 2, 42) = 90.76, p < .001, q 2 = .812 for number of thought units. 
As Table 7 shows, participants in the retrospective condition verbalized the fewest number of 
words (M= 658.80, SD = 229.00) and thought units (M= 30.73, SD = 13.15). Participants in the 
concurrent condition verbalized more than twice the number of words as participants in the 
retrospective condition (M = 1676.13, SD = 394.89) and more than four times the number of 
thought units than participants in the retrospective condition (M = 147.40, SD = 35.83). 
Participants in the virtual revisit condition verbalized the most number of words (M = 2637.87, 
SD = 359.94). However, participants in the virtual revisit condition verbalized less thought units 
than teachers in the concurrent condition (M = 123.27, SD = 20.58). This finding suggests that 
thought units verbalized by participants in the virtual revisit condition contained more words 
than thought units verbalized by participants in the concurrent condition. 

Table 7 


ANOVA Summary Statistics for Word Count and Number of Thought Units 



Concurrent 

Condition 

Retrospective 

Condition 

Virtual 

Revisit 

Condition 





M 

iSD) 

M 

iSD) 

M 

iSD) 

F 

P 

r 

Word Count 

1676.13 

(394.89) 

658.80 

(229.00) 

2637.87 

(359.94) 

130.421 

.000** 

.861 

Number of Thought 
Units 

147.40 

(35.83) 

30.73 

(13.15) 

123.27 

(20.58) 

90.756 

.000** 

.812 


**p < .001 

Note: df= (2, 42) for all variables 


Post hoc comparisons using Tukey HSD revealed significant differences in the frequency 
of words ip < .01) and the frequency of thought units ip < .05) between all three conditions. In 
contrast to the participants in the concurrent and virtual revisit conditions who were asked to 
verbalize their thoughts for 20 minutes, participants in the retrospective condition were not given 
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a time constraint to verbalize their thoughts. Although an unlimited amount of time was given to 
participants in the retrospective condition, these participants verbalized their thoughts for an 
average four minutes 30 seconds. Therefore, it is not surprising that participants in the 
retrospective condition verbalized significantly fewer words and thought units than the other two 
conditions. A more interesting finding is the difference in word counts and number of thought 
units between the concurrent and virtual revisit conditions. Participants in the virtual revisit 
condition verbalized significantly more words than participants in the concurrent condition; 
however, the reverse is true for the number of thought units. This finding indicates that thought 
units produced by participants in the virtual revisit condition contained a greater number of 
words. This suggests that the thought units verbalized by participants in the virtual revisit 
condition were more complex than the thought units verbalized by participants in the concurrent 
condition. For example, as a participant from the concurrent condition views the homepage for 
the first time she describes her procedural behavior: “I’m going to have a look at the How To 
Videos.” A participant from the virtual revisit condition who also views the homepage for the 
first time goes further to provide a reason for her behavior: “I’m always interested in ways to 
increase my students’ background knowledge of vocabulary and comprehension so I go back and 
forth between comprehension and vocabulary before I narrowed it down and selected 
vocabulary.” Both of these thought units were verbalized during the participants’ initial view of 
the home page. However, the thought unit verbalized by the participant in the virtual revisit 
condition is more complex in that it provides a reason for her navigational choice. While 
participants in the concurrent condition verbalized on average more thought units, the 
verbalizations were less likely to include reasons for their decisions. Examples of thought units 
produced by participants in the concurrent condition include: that’s interesting; let’s see what 
that is; there’s a word wall; that’s like what we did in kindergarten. 

Pearson correlations. Pearson correlations were computed to detennine three 
relationships: between the cognitive processes, between the frequency of word counts and 
thought units, and between the cognitive processes and word counts and thought units (see Table 
8). According to the results, higher order cognitive processes (planning, connecting, reasoning, 
and reflecting) were positively correlated to other higher order cognitive processes, and 
negatively correlated to lower order cognitive processes (diversion, understanding, describing 
website features, and reading). Similarly, lower order cognitive processes were more likely to be 
positively correlated to other lower order cognitive processes and negatively related to higher 
order cognitive processes. For instance, reading was negatively correlated to planning (r = -.42, 
y? < .01), connecting (r = -.39, p < .01), reasoning (r = -.50, p < .01), reflecting (r = -.55, p < ■01), 
evaluating website content (r = -.31, p < .05), and evaluating user experience (r = -.37, p <.05). 
In contrast, reading was positively related to diversion ( r = .46 ,p < .01), understanding (r = .30, 
p < .05), and describing website features (r = .45, p < .01). These findings corroborate the above 
ANOVA results and suggest that participants in the concurrent condition, who were more likely 
to read text during the think aloud, were less likely to verbalize thought units related to higher 
order cognitive processes. 

Pearson correlations also revealed a positive correlation between the frequency of words 
and the frequency of thought units (r = .70, p < .01). This finding indicates that as the number of 
words increased the number of thought units increased. Results also revealed a negative 
correlation between higher order cognitive processes and the number of thought units verbalized. 
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This suggests that teachers who verbalized thought units related to higher order processes were 
more likely to verbalize fewer thought units than teachers who verbalized thought units related to 
lower order processes. 


Table 8 

Pearson Correlations 



1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

Cognitive 

Processes 

1. Planning 

1 











2. Connecting 

.31* 

1 










3. Reasoning 

.36* 

.06 

1 









4. Reflecting 

.19 

.38* 

-.10 

1 








5. Evaluating 

-.23 

.03 

.03 

.23 

1 







Website 












6 . Evaluating 
Experience 

.05 

-.08 

-.09 

.16 

-.34* 

1 






7. Diversion 

-.36* 

_ 39** 

-.22 

_ 46** 

-.17 

-.14 

1 





8 . Understanding 

-.26 

-.35* 

-.21 

-.29 

-.34* 

.05 

.38* 

1 




9. Describing 
Behaviour 

-.12 

-.20 

.03 

-.28 

-.56** 

_ 47 ** 

-.003 

.01 

1 



10. Describing 

-.43** 

-.33* 

-.45** 

-.31* 

-.13 

-.23 

39 ** 

.23 

.10 

1 


Website 

11. Reading 

-.42** 

_ 39 ** 

-.50** 

-.55** 

-.31* 

-.37* 

.46** 

.30* 

.18 

.45** 

1 


*p < .05, **p < .01 


Discussion 

The findings of this study indicated significant differences in participants’ verbalizations 
across the three think aloud conditions. One possible reason for these differences relates to 
cognitive load. Cognitive load refers to the amount of mental effort in working memory: 
cognitive load “represents the load that perfonning a particular task imposes on the cognitive 
system” (Pass & Van Merrienboer, 1994, p. 122). Cognitive load may have impacted 
participants in the concurrent condition who were required to use a higher level of mental energy 
to process the given information as they simultaneously completed the website task (Ping Lim, 
2004). The cognitive load on working memory may have diminished the quality of their 
verbalizations. This is consistent with Ericsson and Simon (1993) who state that concurrent 
reporting may become difficult to maintain under high cognitive load conditions. Cognitive load 
research also indicates that during multimedia tasks, concurrent reporting will interfere with 
infonnation processing and limit the extent of the thought units (Nielsen, Clemmensen, & 
Yssing, 2002; van Gog et ah, 2009). As the participants in the concurrent condition verbalized 
their thoughts while simultaneously completing the website task, the cognitive demand on their 
working memory increased: their brains prioritized infonnation processing over verbalizations. 
Therefore, participants in the concurrent condition verbalized fewer thought units related to 
higher cognitive processes (Cooke, 2010). 

In contrast, participants in the retrospective and virtual revisit conditions had the 
advantage of thinking aloud after the website task had been completed; thus, the cognitive load 
on their working memory was lighter during their think aloud than those in the concurrent 
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condition. As a result of a lighter cognitive load, participants in the retrospective and virtual 
revisit conditions had more cognitive resources available during the think aloud; therefore, they 
could focus their mental energy on constructing and verbalizing complex ideas. 

For example, planning is a complex cognitive activity and involves visualizing the future, 
producing and generating new information, and “putting elements together to form an original 
product” (Krathwohl, 2002, p. 215). Planning is also an indicator of teachers’ intentions for 
practice (Clark & Peterson, 1986; Krathwohl, 2002). The planning thought units verbalized by 
participants in the retrospective and virtual revisit conditions contained complex information that 
demonstrate intentions for practice. For instance, a video on social studies and writing 
integration led a participant in the retrospective condition “to think of a different idea that [she] 
could bring into [her] class which includes doing cross-curricular work with probability and 
ancient civilizations and trading cards and games.” Similarly, a teacher in the virtual revisit 
condition described how she could modify a lesson to include cross-curricular integration: 
“Building interviewing and report writing skills, I know that there is a lot that I can do with this 
in tenns of reading and writing and oral and drama and themes like social justice and history and 
so on.” These examples demonstrate complex verbalizations; teachers are planning as they use 
the professional development website—they are visualizing the future and beginning to generate 
new ideas. 

Another possible reason for the findings can be drawn from research on infonnation 
processing and memory recall. Information processing is enhanced when new incoming 
information is connected to prior knowledge and previous experiences (Mastin, 2010; Weber, 
Corrigan, Fomash, & Neupauer, 2003). Information processing is also enhanced when new 
material is interesting to the learner (Gamer & Gillingham, 1991). The more deeply new 
information is processed (i.e., through connections and interest), the more likely it will be 
recalled. In contrast to participants in the concurrent condition, participants in the retrospective 
condition were able to process infonnation during the website task on a much deeper level 
because they did not have the same cognitive demands of verbalizing their thoughts while 
simultaneously completing the website task. They were able to verbalize significantly more 
connections with past experiences and interests than participants in the concurrent condition. For 
instance, a participant in the retrospective condition connected components of the website to her 
current reading program: 

After visiting a virtual tour, I was interested in going through all the different parts of the 

balanced literacy and I clicked on a couple of things that interested me and that I’m 

working on, like comprehension skills and fluency and word building. 

In addition, participants in the retrospective condition did not have access to a visual cue 
or memory aid. It is possible that in the absence of a visual cue, participants in the retrospective 
condition verbalized more connecting thought units because they recalled meaningful memories 
established during the website task. For instance, a participant in the retrospective condition 
connected website content on hand writing to a student in her class: “I saw something about hand 
writing which made me think about a student that I have in grade one who...it’s quite a struggle 
for her to read what she’s writing.” The information that this participant recalled was 
meaningful to her because it directly related to the needs of a student in her classroom. 
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While the retrospective procedure may produce verbalizations related to meaningful 
connections, the retrospective procedure is limited by the fact that resulting verbalizations are 
based on the ability to recall infonnation. Participants in the retrospective condition of the 
current study omitted most of their navigation, particularly the intermediate web-based 
behaviors. In contrast, participants in the virtual revisit condition had direct access to their web- 
based actions via the screen-capture recording. The screen recording captured participants’ 
website navigation and acted as a visual aid during the think aloud. One possibility is that 
participants in the virtual revisit condition utilized the visual infonnation as an aid to recall their 
navigational decisions and why they made them. Available cognitive resources and direct access 
to web-based actions allowed participants in the virtual revisit condition to extend their 
descriptions and clarify the reasons for their navigational choices. While the screen recording 
could have acted as a memory aid, it is also possible that the screen recording prompted 
participants to generate rationalizations for their decisions during the think aloud. In any case, 
the virtual revisit allowed participants to produce thorough verbalizations related to their 
navigational decisions and why they made them. Reasoning about behavior moves beyond 
simple descriptions of actions and offers rich explanations about decisions. For example, a 
participant from the virtual revisit condition provided a descriptive rationale for her decision to 
remain on a particular webpage for an extended period of time: “At this point I was trying to just 
read what the student wrote to get an idea of whether they were creating their own stories or 
whether they were doing more of a retell.” Reasoning thought units provide thorough 
descriptions, clarifications, extensions and overall greater insight into participants’ navigational 
choices—participants explain why they make particular navigational decisions. 

Overall findings from this study reveal benefits and limitations to employing each type of 
think aloud method in the context of a professional development website. A benefit of the 
concurrent think aloud is that it generates direct data about the ongoing cognitive processes that 
occur during task perfonnance. Since the two activities, thinking aloud and task perfonnance 
occur simultaneously the verbalizations are valid fonns of infonnation—the verbalizations 
contain direct data about participants’ thoughts in real time. The limitations of the concurrent 
method, however, may outweigh this benefit. The first limitation of the concurrent think aloud 
method is the fact that this think aloud produces fewer verbalizations related to higher order 
cognitive processes. Secondly, the concurrent think aloud method results in a high cognitive 
load on working memory. 

The main benefit of employing the retrospective procedure is that this method produces 
verbalizations related to higher cognitive processes (planning, connecting, and reflecting). 
However, the limitations of employing the retrospective think aloud method are significant. 
Participants will omit most of their online actions and navigational decisions. Furthermore, 
participants will most likely have difficulty recalling their intennediate web-based actions and 
reasons for these decisions. 

The main benefit of employing the virtual revisit procedure is that this method produces 
verbalizations related to higher level cognitive processes (planning, reasoning, and reflecting). 
Secondly, participants can rely on a visual aid to help them recall their navigational decisions 
and why they made them. One limitation of the virtual revisit think aloud is the time required to 
complete both the website task and think aloud task. In the current study, the total time for each 
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participant in the virtual revisit condition was 40 minutes. For some researchers, this may be 
costly. Moreover, the time required to employ the virtual revisit think aloud may limit 
participant involvement. 

Collectively, the findings indicate that the virtual revisit can avoid the limitations of the 
concurrent and retrospective procedures and provide thorough and descriptive thought units and 
insights into how teachers use and leam from a professional development website. 

Study Limitations 

There were three main limitations to this study: (1) factors that may have caused 
reactivity, (2) the use of one professional development website, and (3) the possibility of 
researcher bias. First, factors may have caused reactivity during the think aloud procedure. 
Reactivity occurs when task perfonnance is altered as a result of an awareness of the study task. 
Reactivity may have occurred as a result of participants’ awareness that they were completing a 
task in the presence of the primary investigator. A “motivational shift” in which the participants 
anticipate exposure of their think aloud protocol may have occurred when the participants were 
informed of the think aloud procedure (Russo, Johnson, & Stephens, 1989). Another factor that 
may have caused reactivity is hearing one’s own voice. The additional aural stimulation may 
have interfered with the concurrent navigation (Russo et ah, 1989). In general, reactivity was 
reduced as much as possible during the one-on-one meeting by: staying neutral during the task 
and think aloud, keeping verbal and nonverbal cues to a minimum, and providing participants in 
the retrospective and virtual revisit conditions with the second part of the instructions after they 
had completed the website task. However, the factors described above should be considered 
when interpreting the findings. 

Secondly, the current study was context-specific and used one professional development 
website. Future research comparing the three think aloud conditions should be conducted with 
additional websites and online resources. Conducting similar studies with alternative 
professional development websites will enhance the credibility and transferability of the results. 

Finally, it is difficult to eliminate researcher bias. Steps were taken to reduce researcher 
bias as much as possible: the use of relevant literature to develop the coding scheme, unitizing 
the transcripts based on Lincoln and Guba’s (1985) procedures, using a second coder procedure, 
and staying as close to the data as much as possible during the analysis and interpretation of the 
findings. However, this limitation must be considered when interpreting the findings. To avoid 
researcher bias, future research could involve a team of researchers with varying backgrounds, 
particularly during data analysis. 

Significance and Educational Implications 

The significance of this research is that it compares the effectiveness of two traditional 
think aloud methods—concurrent and retrospective—with a think aloud method combining a 
retrospective procedure with screen capture technology (the virtual revisit). While think aloud 
research is extensive, studies that compare the effectiveness of different think aloud 
methodologies for understanding cognitive processes as website users navigate online resources 
are scarce. Based on the findings of this study, there may be potential benefits and limitations to 
employing each type of think aloud method. In addition, the virtual revisit think aloud, a 
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relatively underused type of think aloud method, appears to be an effective method for 
examining teachers’ cognitive processes as they use a professional development website. The 
virtual revisit think aloud method produces thorough verbalizations that incorporate reasons 
behind the decision-making process. 

Another significance of the current study is that the participants were practicing teachers 
with between one and over 25 years of classroom experience. Much of the research examining 
teachers’ beliefs and attitudes towards learning in online environments has studied preservice 
teachers. Studying the cognitive processes of practicing teachers has the potential to contribute 
to the understanding of teacher professional development and teacher cognition. 

Furthermore, the virtual revisit think aloud method also has potential that transcends the 
specific domain in which it is applied in the current study. Virtual revisits with think aloud could 
be applied to examine cognitive processes of participants in research involving online learning 
and website use in many domains in education. If researchers provide feedback to educational 
website developers based on the virtual revisit methodology, improvements can be made to the 
design and content of their sites. 


Conclusion 

Teachers learn in many different aspects of practice, including their classrooms, their 
school communities, professional development courses, and online environments (Borko, 2004). 
To understand teacher learning, it must be studied within these multiple contexts and it must be 
studied with an effective methodology that provides rich and thorough data about the reasoning 
process. Gaining greater insights into teachers’ cognitive processes as they navigate online 
enviromnents can lead to the reconsideration of the design of online learning environments so 
that they “are more conducive to infonnal learning...so that they further develop the ability of 
professionals to solve problems and learn independently” (Lohman, 2006, p. 144). 
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